Repository navigation
Parallelize test suite to speed up CI - #578
Conversation
skarim
changed the title
skarim
changed the title
skarim
changed the title
There was a problem hiding this comment.
🔵 Needs a closer look
The new process-tree cancellation and CI completion checks need final human review backed by successful native-platform runs on Linux, Windows, and macOS.
No open issues 🎉
What changed in this PR
This PR introduces a process-isolated test runner to shorten gh-stack’s cross-platform test suite while retaining race detection and checking test completion.
Changes:
- Adds test discovery, weighted scheduling, bounded execution, and result verification.
- Shards Windows and macOS CI and adds a cross-platform completion gate.
- Streamlines fixtures, restores shared test state, and updates testing documentation.
| File | Description |
|---|---|
scripts/test/timings.go |
Adds advisory scheduling weights for expensive tests. |
scripts/test/schedule.go |
Assigns tests to partitions with bounded selectors. |
scripts/test/report.go |
Validates results and consistency across shard reports. |
scripts/test/process_windows.go |
Adds Windows process-tree cancellation. |
scripts/test/process_unix.go |
Adds Unix process-group cancellation. |
scripts/test/main.go |
Implements discovery, race-enabled builds, execution, and reporting. |
scripts/test/main_test.go |
Tests scheduling, reports, workflow configuration, and process deadlines. |
README.md |
Links to contributor instructions. |
internal/modify/apply_test.go |
Splits action tests for scheduling and batches fixture queries. |
internal/git/rebase_start_test.go |
Uses a lighter local repository fixture. |
internal/git/gitops_test.go |
Streamlines fixtures and enables scoped tests to run in parallel. |
CONTRIBUTING.md |
Documents runner usage, artifacts, and CI verification. |
cmd/view_test.go |
Restores Git mocks and checks cleanup. |
AGENTS.md |
Updates testing commands and CI guidance. |
.github/workflows/test.yml |
Adds shards, artifacts, a completion gate, and stacked-PR triggers. |
.github/copilot-instructions.md |
Updates runner guidance and test-isolation precautions. |
Review effort: Balanced
| push: | ||
| branches: [main] | ||
| pull_request: | ||
| branches: [main] |
There was a problem hiding this comment.
❓ Are we dropping main as a branch argument here to be more permissive so that CI is run for any and all base branches, not solely main anymore? Asking to confirm intent!
On the one hand, I can see this being beneficial for stacked PRs within this repo, like the companion PR to follow this: #579
On the other hand, possibility of increasing compute cost? 😅
There was a problem hiding this comment.
Yeah I figured might as well make it more permissive so it runs on all PRs, stacked or not. No particular need for this tbh, but can add the restriction back if CI compute gets out of hand.
| | 7 | Rebase already in progress | | ||
| | 8 | Stack is locked by another process | | ||
|
|
||
| ## Contributing |
There was a problem hiding this comment.
CONTRIBUTING.md is present, but never hurts to be explicit in the README itself
francisfuzz
left a comment
francisfuzz
left a comment
There was a problem hiding this comment.
Approved! ✅ This meaningfully reduces the wait for CI. I compared the latest run against the 10 Tests workflow runs immediately before this PR.
| Metric | Previous median | Latest PR run | Change |
|---|---|---|---|
| Full workflow | 23m 45s | 11m 56s | 50% shorter |
| Windows tests | 22m 36s | 10m 23s | 54% shorter |
| macOS tests | 8m 15s | 3m 19s | 60% shorter |
| Linux tests | 1m 06s | 1m 28s | 33% longer |
Platform test times use the slowest shard.
The two latest revisions both finished in about 12 minutes. The tradeoff is higher aggregate runner usage from parallelizing Windows and macOS, while Linux gets slightly slower. Saving about 12 minutes per CI cycle makes that a worthwhile tradeoff to me.
Comparison details
I used the 10 runs preceding this PR because three of the latest repository runs belong to #579 and already inherit this setup. The baseline includes one failed run. These are observed hosted-runner timings across different commits, not a controlled benchmark.
Refactors the tests and adds a dedicated runner to speed up the suite through parallel execution.
Functionality and impact
Key areas to review